Forward-LASSO with Adaptive Shrinkage

Both classical Forward Selection and the more modern Lasso provide computationally feasible methods for performing variable selection in high dimensional regression problems involving many predictors. We note that although the Lasso is the solution to an optimization problem while Forward Selection is purely algorithmic, the two methods turn out to operate in surprisingly similar fashions. Our results demonstrate, both empirically and theoretically, that neither procedure dominates the other. We propose a new method we call Forward-Lasso Adaptive SHrinkage (FLASH), which incorporates both Forward Selection and the Lasso as special cases. FLASH works well in situations where either Forward Selection or the Lasso dominates but also performs well in situations where neither method succeeds. FLASH is fitted using a variant of the computationally efficient LARS algorithm. We provide an extensive theoretical analysis showing that many of the error bounds that have recently been developed for the Lasso can be improved using FLASH. Finally we demonstrate, through numerous simulations and a real world data set, that FLASH generally outperforms many competing approaches. Some key words: Forward Selection; Lasso; Shrinkage; Variable Selection